Hardware-in-the-loop simulation system and method for computer vision

ABSTRACT

The disclosure relates to a hardware-in-the-loop simulation system and method for computer vision. An embodiment of the disclosed system comprises a software simulation and a hardware simulation. The software simulation includes a virtual scene and an observed object that are generated by virtual reality software. The virtual scene images are obtained at different viewpoints. The hardware simulation includes the virtual scene images being projected onto a screen by a projector, wherein the projected scene images are shot by a camera, and where in the direction of the camera is controlled by a pan-tilt.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Chinese Patent Application SerialNos. 200610083637.7 filed May 31, 2006 and 200610083639.6 filed May 31,2006, the disclosures of which, including their specifications, drawingsand claims, are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to a hardware-in-the-loop simulation forcomputer vision. In particular, the disclosure relates to a system andmethod for performing a hardware-in-the-loop simulation. The systemintegrates an actual camera into virtual reality (VR) simulationenvironment to provide a hardware-in-the-loop simulation platform forthe development and verification of a computer vision algorithm ormethod.

BACKGROUND

As a new interdisciplinary field, computer vision has made greatprogress in both theoretical research and practical application.Especially, following the development of image processing technology andcomputers with high performance in recent years, computer vision hasbeen applied in many practical applications, such as targetidentification, surveying and mapping, industrial measurement,industrial control, automatic navigation and military application, etc.

The simulation of camera imaging can provide a convenient and quickexperiment platform for the study of computer vision to get simulationdata under various conditions, which saves money and shortens the timefor study. Therefore, it is significant in the process of the study ofthe computer vision. Especially, when the actual operating conditionsare special, the simulation system becomes more important, such asaerial mapping which is to get realtime 3D terrain data but where acomputer vision algorithm cannot be carried under the actual flightenvironments based on cost or time constraints.

At present, the simulations of computer vision are almost digitalsimulations. Although complete software simulations could reduce thecost and time of study and experimentation to a minimum level, it isdifficult to establish mathematical models for complex computer visionsystems. The complex systems have to be simplified for complete softwaresimulations, so disparities exist between the conditions of the completedigital simulation and the actual working condition because of excessiveidealization. Especially when the camera exhibits lens distortion andstronger noises interfere, the truthfulness and reliability of imagesacquired by complete digital simulation become bad. Hardware-in-the-loopsimulations can overcome above insufficiency. Hardware-in-the-loopsimulation systems replace the mathematical model of the performancehardware with the hardware itself in order to approach the actualcondition for a more accurate simulation result. Thesehardware-in-the-loop systems are economical means of computer visionstudy and experiment under various conditions, such as the utilizationto testify the correctness and evaluate the performance of computervision algorithm in the initial stage of study.

The hardware-in-the-loop simulation system has to integrate softwarewith hardware. It is rather difficult to establish the system modelbetween software and hardware to determine the relationship between thevirtual reality scene and the real-life camera image, so it is abottleneck of computer vision hardware-in-the-loop simulation. There isno hardware-in-the-loop simulation system or method to solve the aboveproblem for computer vision mentioned in the existing literatures atpresent. So it is necessary and urgent to solve the bottleneck problemand set up a practical and efficient hardware-in-the-loop simulationsystem for computer vision.

SUMMARY

The present disclosure is directed to solving the above describedproblem and provides a hardware-in-the-loop simulation system and methodfor computer vision. It integrates actual camera into VR simulationenvironment, to construct the sensor structure integrated hardware andsoftware for hardware-in-the-loop simulation according to cascadeprojection transformations. It possesses the advantages of highaccuracy, simple structure and low cost.

One embodiment of the disclosed hardware-in-the-loop simulation systempresented herein comprises a VR imaging unit, a projection unit and anactual camera unit. The three units are sequential and their positionsare fixed with respect to each other. The VR imaging unit builds updesired objects or scenes corresponding to the actual environment andproject rendered objects or scenes onto the display as a virtual cameraby using the VR technology, e.g. OpenGL. The projection unit projectsthe virtual scene image, i.e. virtual camera image, on the computerdisplay onto the projection screen by a projector. The actual cameraunit acquires the image of scene on the projection screen taken by anoff-the-shelf camera.

Based on the above-described apparatus it is possible to achieve ahardware-in-the-loop simulation according to the following method.Firstly set up virtual objects or scenes by using the VR software, setimaging parameters to acquire the 3D image of the virtual scene indesired direction. Then project the computer virtual scene image ontothe screen by projector. Lastly, acquire an output image by shooting thescene image on the screen with actual camera. The camera image is theresult of the simulation as a computer vision experiment object. Beforethe experiment data processing and analysis the parameters of virtualcamera should be calculated, the actual camera and projector modelsshould be calibrated for the parameters estimation. Comparing thecomputer vision arithmetic experiment result based on the simulationimages and the parameters of system with the known character data of thevirtual objects, it can be testified the correctness and evaluate theperformance of the algorithm.

The present disclosure builds a hardware-in-the-loop simulation platformwhich integrates software with hardware and has the followingadvantages. First, VR technology is integrated with computer visiontechnology creatively and effectively. The system not only has theadvantages of easier realization and convenient operation as softwaresimulation, but also has the advantage of high similarity to the actualexperiment conditions as hardware simulation. It fills the gap of thehardware-in-the-loop simulation technology for computer vision. Second,the structure of system is simple and cost is low, because main parts ofthe system are a common projector, and off-the-shelf camera andcomputers. It is easily achieved and fits a wide variety of applicationsin the computer vision field, so the system is worth to be popularized.Finally, the precision of system is rather high to ensure the validityof simulation. The result of stereo vision 3D reconstruction simulationindicates that the measure precision of 3D reconstruction based on thehardware-in-the-loop simulation system can reach 1%.

Other objects and features of the present invention will become morereadily apparent from the following detailed description of thepreferred embodiment taken in conjunction with the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of specification, illustrate an exemplary embodiment of the presentinvention and, together with the general description given above and thedetailed description of the preferred embodiment given below, serve toexplain the principles of the present invention.

FIG. 1 illustrates a hardware configuration of a hardware-in-the-loopsimulation system according to an embodiment of the present invention;

FIG. 2 illustrates three projection transformation models of ahardware-in-the-loop simulation system for computer vision;

FIG. 3 illustrates an embodiment of a linear pinhole camera model with acamera coordinate system and a world coordinate system;

FIG. 4 is a flowchart of a stereo vision simulation processing using thehardware-in-the-loop simulation system in accordance with an embodimentof the present invention;

FIG. 5 illustrates the geometry of the pinhole camera with perspectiveprojection;

FIG. 6 illustrates an initial default state of a virtual camera, i.e.the initial default relationship between the virtual world coordinatesystem and the virtual camera coordinate system;

FIG. 7 illustrates a calibration pattern of an embodiment.

DETAILED DESCRIPTION

While the claims are not limited to the illustrated embodiments, anappreciation of various aspects of the present invention is best gainedthrough a discussion of various examples thereof. Referring now to thedrawings, illustrative embodiments will be described in detail. Althoughthe drawings represent the embodiments, the drawings are not necessarilyto scale and certain features may be exaggerated to better illustrateand explain an innovative aspect of an embodiment. Further, theembodiments described herein are not intended to be exhaustive orotherwise limiting or restricting to the precise form and configurationshown in the drawings and disclosed in the following detaileddescription.

The system is used for a hardware-in-the-loop simulation of stereovision. First, a description will be given below of a configuration of ahardware-in-the-loop simulation system of computer vision. FIG. 1illustrates the hardware configuration of the hardware-in-the-loopsimulation system. The system comprises a projector 1, a camera 2, apan-tilt 3 for camera 2, a projection screen 4, and computers 5 and 6.The projector 1 is fixed upon a ceiling 7 such as, for example, with ahanger. The pan-tilt 3 and projection screen 4 are fixed to the wall bysteady brackets or on the wall 8 directly. The projector 1 may be acommon off-the-shelf product and is configured to communicate with thecomputer 5 in which virtual reality, hereinafter “VR”, software isinstalled for generating virtual 3D scene images. The virtual sceneimage is projected onto the screen 4. The camera 2 may be anoff-the-shelf CCD camera which is fixed on the pan-tilt 3. The camera 2is configured to communicate with the computer 6 with a frame grabberboard. The computer 6 is also used to control the pan-tilt 3 for asuitable direction of camera to select a proper “FOV” (field of view) ofcamera 2. After positioning of camera 2 is determined, the pan-tilt 3should not rotate any more during the use of the system. The principleof selecting lens of camera 2 is that the size of FOV of camera 2 is assame as possible as the size of image on the projection screen 4 and notlarger than the image's. The parameters of the system should becalculated and calibrated for application of the simulation system, forexample, the camera 2 should be calibrated in the computer vision. Whilethe system is in the parameters calibration or work state, the positionsand parameters of the projector 1, the camera 2, the pan-tilt 3 and thescreen 4 must be fixed for high precision of simulation.

Referring to FIG. 2, in one embodiment, the system is comprised of threeprojection transformation models. These models include virtual cameramodel M1, projector imaging model M2 and camera model M3. The threemodels are operatively connected with each other in series, so that theoutput of one model is the input of another. The virtual camera model M1is realized by VR software such as, for example, OpenGL. The virtualcamera images under various conditions are acquired through settingcorresponding parameters. The virtual camera model is a lineartransformation, and the parameters of virtual camera can be calculatedaccording to the function parameters of VR software. The projectorimaging model M2 and the camera model M3 are realized by actualapparatuses; they are nonlinear plane-to-plane projectiontransformations. The hardware parts of the hardware-in-loop simulationsystem should be calibrated.

In this embodiment, the pinhole camera model that most algorithms ofcomputer vision usually are based on is used. The virtual camera modelis an ideal linear pinhole camera model. Referring to FIG. 3, the lineartransformation from the world coordinate to the camera coordinate isdescribed by the following equation,

$\begin{matrix}{\left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack} & \; \\{{s_{v}\begin{bmatrix}u_{v} \\v_{v} \\1\end{bmatrix}} = {{{\begin{bmatrix}\alpha_{vx} & 0 & u_{v\; 0} & 0 \\0 & \alpha_{vy} & v_{v\; 0} & 0 \\0 & 0 & 1 & 0\end{bmatrix}\begin{bmatrix}R_{v} & T_{v} \\0^{T} & 1\end{bmatrix}}\begin{bmatrix}X_{vw} \\Y_{vw} \\Z_{vw} \\1\end{bmatrix}} = {{M_{1v}{M_{2v}\begin{bmatrix}X_{vw} \\Y_{vw} \\Z_{vw} \\1\end{bmatrix}}} = {M_{v}\begin{bmatrix}X_{vw} \\Y_{vw} \\Z_{vw} \\1\end{bmatrix}}}}} & (1)\end{matrix}$

Where (X_(vw), Y_(vw), Z_(vw), 1)^(T) is the 3D world homogeneouscoordinate with the subscript v indicating the virtual camera model,(u_(v), v_(v), 1)^(T) is the computer image homogeneous coordinate,α_(vx)=f_(v)/dx_(v) and α_(vy)=f_(v)/dy_(v) are the scale factors in thedirection of X and Y axes of the virtual camera image, f_(v) iseffective focal length of virtual camera, dx_(v) and dy_(v) are thedistance between adjacent sensor elements in X and Y directionsrespectively, (u_(v0), v_(v0)) is the coordinate of the principal point,s_(v) is an arbitrary scale factor, R_(v) and T_(v) are the 3×3 rotationmatrix and translation vector which relate the world coordinate systemto the camera coordinate system. M_(1v) is called the camera intrinsicmatrix is determined by the intrinsic parameters α_(vx), α_(vy), u_(v0)and v_(v0), and M_(2v) is called the camera extrinsic matrix isdetermined by the extrinsic parameters R_(v) and T_(v). M_(v) is calledthe projection matrix of which the elements can be acquired according tothe parameters of VR software.

Like the virtual camera model, the projector imaging model is describedby the following equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 2} \right\rbrack & \; \\{\;{{s_{p}\begin{bmatrix}u_{p} \\v_{p} \\1\end{bmatrix}} = {{{\begin{bmatrix}\alpha_{px} & 0 & u_{p\; 0} & 0 \\0 & \alpha_{py} & v_{p\; 0} & 0 \\0 & 0 & 1 & 0\end{bmatrix}\begin{bmatrix}R_{p} & T_{p} \\0^{T} & 1\end{bmatrix}}\begin{bmatrix}X_{pw} \\Y_{pw} \\Z_{pw} \\1\end{bmatrix}} = {M_{p}\begin{bmatrix}X_{pw} \\Y_{pw} \\Z_{pw} \\1\end{bmatrix}}}}} & (2)\end{matrix}$

The definitions of the parameters in the above equation (2) are the sameas those in equation (1) with the subscript p indicating the projectorimaging model. Without loss of generality, it can be assumed thatZ_(pw)=0 because of the plane-to-plane projection transformation. Sofrom equation (2), we have:

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack & \; \\\begin{matrix}{\mspace{79mu}{{s_{p}m_{p}} = {{\begin{bmatrix}\alpha_{px} & 0 & u_{p\; 0} \\0 & \alpha_{py} & v_{p\; 0} \\0 & 0 & 1\end{bmatrix}\left\lbrack {r_{1p}\mspace{14mu} r_{2p}\mspace{14mu} T_{p}} \right\rbrack}\begin{bmatrix}X_{pw} \\Y_{pw} \\1\end{bmatrix}}}} \\{= {H_{p}{\overset{\sim}{X}}_{pw}}}\end{matrix} & (3)\end{matrix}$

Where m_(p)=[u_(p) v_(p) 1]^(T), r_(1p), r_(2p) are the first and secondcolumns of R_(p), {tilde over (X)}_(pw)=[X_(pw) Y_(pw) 1]^(T).

Likewise, the equation of the actual camera model can be deduced throughthe same process as the projector imaging model. It is described by thefollowing equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack & \; \\{{s_{c}\begin{bmatrix}u_{c} \\v_{c} \\1\end{bmatrix}} = {{s_{c}m} = {{{\begin{bmatrix}\alpha_{cx} & 0 & u_{c\; 0} \\0 & \alpha_{cy} & v_{c\; 0} \\0 & 0 & 1\end{bmatrix}\left\lbrack {r_{1c}\mspace{14mu} r_{2c}\mspace{14mu} T_{c}} \right\rbrack}\begin{bmatrix}X_{cw} \\Y_{cw} \\1\end{bmatrix}} = {H_{c}{\overset{\sim}{X}}_{cw}}}}} & (4)\end{matrix}$

The definitions of the parameters in the above equation (4) are the sameas those in the equation (3) with the subscript c indicating the cameramodel.

The lens distortions of the camera and projector are not considered inequations (3) and (4). But a real-life camera usually exhibits lensdistortion. So a nonlinear camera calibration technique should beadopted to calibrate the parameters or hardware part of the system, andthe process of the calibration will be expounded in detail in thefollowing section.

Referring to FIG. 4, in one embodiment, there are five steps to achievethe stereo vision hardware-in-the-loop simulation. First, the virtualobject or scene is generated by VR software and the virtual scene imagesare acquired at two or several different viewpoints by setting thefollowing parameters, the vertical FOV angle in degree θ, the aspectratio of the width to height of the viewport (computer screen) Aspect,the distance from viewpoint to near and far clipping planes NearPlaneand FarPlane, the position of viewpoint i.e. the 3D coordinates in theworld coordinate system XPos, YPos, ZPos, and the yaw, pitch and rollangles θ_(Yaw), θ_(Pitch), θ_(Roll). Second, the virtual scene imagesare projected onto the screen 4, respectively, by the projector 1.Third, the virtual 3D scene images of different viewpoints on screen 4are respectively shot using the actual camera 2 to acquire the outputimages of the simulation system. Fourth, the parameters of virtualcamera are calculated, and the actual camera and projector models arecalibrated for high-accuracy parameters estimation. Fifth, thecoordinates of virtual objects or scenes in the world coordinate systemare obtained by using basic principle and correlative arithmetic of thestereo vision according to the camera images to confirm the correctnessand evaluate the performance of the stereo vision algorithm.

The third and fourth steps in the FIG. 4 are the most important forsimulation, because the correctness of simulation result depends on thevalidity of the parameters of the system that are calculated andcalibrated. The process how to acquire the parameters of system isdescribed in detail in the following text.

First, the intrinsic parameters of the virtual camera will be calculatedaccording to the parameters of the projection transformation of the VRsoftware, the FOV angle θ and the image resolution. Referring to FIG. 5,the following equation of trigonometric function may be used to describethe relationship between the set projection transformation parameter andthe intrinsic parameters of virtual camera.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack & \; \\{\mspace{79mu}{{\tan\left( \frac{\theta}{2} \right)} = {\frac{\frac{h}{2}*{dy}_{v}}{f_{v}} = \frac{h}{2\;\alpha_{vy}}}}} & (5)\end{matrix}$

Where h is the height of image in the Y direction in virtual cameraimage plane and its unit is the number of pixels. The scale factorsα_(vy) can be calculated according to the equation (5). The scale factorα_(vx) and α_(vy) in image X and Y axes are equal to each other, and thecoordinate of the principal point (u_(v0), v_(v0)) is the center ofimage, since the virtual camera is ideal pinhole camera. Then theintrinsic parameters matrix of the virtual camera is determined.

The extrinsic parameters of the virtual camera are calculated throughthe coordinate transformations. The extrinsic parameters R_(v) and T_(v)are relative to the parameters of the position of viewpoint and the yaw,pitch and roll angles θ_(Yaw), θ_(Pitch), θ_(Roll) used for model viewprojection of the VR software. The initial default state of the virtualcamera coordinate system is illustrated in the FIG. 6. Referring to FIG.6, the virtual world coordinate system is (X, Y, Z) and the virtualcamera coordinate system is (X_(c0), Y_(c0), Z_(c0)). The viewpoint ofthe virtual camera is located at the origin of the world coordinatesystem, looking along the positive X-axis. So the rotation matrix fromthe world coordinate system to the initial default state of the virtualcamera coordinate system R_(v1) is determined as follows.

$R_{v\; 1} = \begin{bmatrix}0 & 0 & 1 \\0 & {- 1} & 0 \\1 & 0 & 0\end{bmatrix}$

The virtual camera rotates in the order of roll, yaw and pitch from theinitial default state with rotation matrix R_(v2). Then the viewpointmoves from origin to the set position relative to the translation vectorT_(v) to achieve the transformation from the virtual world coordinatesystem to the virtual camera coordinate system. The rotation matrixR_(v2) can be acquired according to the Euler equation as the followingequation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 6} \right\rbrack & \; \\{R_{v\; 2} = \begin{bmatrix}{\cos\;\beta\;\cos\;\gamma} & {\cos\;\beta\;\sin\;\gamma} & {{- \sin}\;\beta} \\{{\sin\;\alpha\;\sin\;\beta\;\cos\;\gamma} - {\cos\;\alpha\;\sin\;\gamma}} & {{\sin\;\alpha\;\sin\;\beta\;\sin\;\gamma} - {\cos\;\alpha\;\cos\;\gamma}} & {\sin\;\alpha\;\cos\;\beta} \\{{\cos\;\alpha\;\sin\;\beta\;\cos\;\gamma} + {\sin\;\alpha\;\sin\;\gamma}} & {{\cos\;\alpha\;\sin\;\beta\;\sin\;\gamma} + {\sin\;\alpha\;\cos\;\gamma}} & {\cos\;\alpha\;\cos\;\beta}\end{bmatrix}} & (6)\end{matrix}$

Where α=θ_(Pitch), β=θ_(Yaw), γ=θ_(Roll). It can be obtained therotation matrix R_(v)=R_(v2)*R_(v1). On the assumption that theviewpoint coordinates in the world coordinate system is T_(vW), thetranslation vector T_(v) can be calculated by the following equation.[Equation 7]T _(v)=−(R _(v) *T _(vW))  (7)

With the above solution procedure, the parameters of virtual camera areacquired. Subsequently, the parameters of the projector imaging model M2and the actual camera model M3 will be acquired by using the followingcalibration method.

In the application of this simulation system, there is no need toestimate the projector imaging model M2 and the actual camera model M3respectively, i.e. there is no need to determine the transformation fromthe projector image to the projection screen and the transformation fromthe projection screen to the camera image respectively. Only thedetermination of the transformation relationship between the projectorimage and the camera image is needed. So the projector imaging model andthe actual camera model are regarded as a whole module for calibration.A nonlinear camera calibration technique is adopted because of the lensdistortions of the camera 2.

A calibration pattern is shown in FIG. 7. The image of a model planecontains a pattern of 4×4 squares, these corners are used forcalibration. The calibration steps of the parameters are described asfollows. First, the center of camera image is determined using a methodof varying focal length presented by Lenz and Tsai (Lenz. R. K, Tsai. R.Y, Techniques for Calibration of the Scale Factor and Image Center forHigh Accuracy 3-D Machine Vision Metrology, IEEE Transactions on PatternAnalysis and Machine Intelligence. Volume 10. Issue 5. September 1988.Pages: 713-720). Then assuming the projector imaging model is a lineartransformation, each four corresponding spatial collinear points on theplanes of the projector computer image, the projection screen image andthe camera coordinate system have the character of cross ratioinvariability. The lens distortion calibration is achieved using theapproach based on cross ratio invariability presented by Zhang Guangjun(“Machine Vision” 2005, written by Zhang Guangjun and published byScientific Publishing Center, Beijing). Finally, based on the center ofcamera image and the lens distortion, the coordinate of the corner incalibration pattern in the camera coordinate system (X, Y) can becalculated according to the image coordinate in pixel (u_(c), v_(c)).The transformation from camera image coordinate (X, Y) to the projectorcomputer image coordinate (u_(p), v_(p)) is linear, which is describedby the following equation.

$\begin{matrix}\left\lbrack {{Equation}\mspace{14mu} 8} \right\rbrack & \; \\{\mspace{79mu}{\begin{pmatrix}u_{p} \\v_{p} \\1\end{pmatrix} = {H*\begin{pmatrix}X \\Y \\1\end{pmatrix}}}} & (8)\end{matrix}$

Where H is a 3×3 matrix. Based on the corners coordinates of calibrationpattern, linear equation groups are set up according to the equation(8). The linear transformation matrix H can be estimated using theleast-squares method.

As described above in detail, all of parameters of hardware-in-the-loopsimulation system are acquired. The output images of simulation systemare regarded as the images of a camera of which the intrinsic andextrinsic parameters have been known according to the parameters ofsimulation system. So the correctness of the system may be confirmed andthe performance of stereo vision algorithm such as feature extraction,stereo matching and 3D reconstruction based on the output images andparameters may be evaluated. The 3D reconstruction precision based onthe hardware-in-the-loop simulation system can reach 1% according to theresults of simulation experiments.

In view of the foregoing, it will be seen that advantageous results areobtained. The hardware-in-the-loop simulation system can be used notonly for stereo vision but also in the field of computer vision for thehardware-in-the-loop simulation of camera imaging of virtual objects orscenes. Furthermore, the system described herein can be used in otherfields for the hardware-in-the-loop simulation of camera imaging.

Although the details of the present invention have been described abovewith reference to a specific embodiment, it will be obvious to thoseskilled in the art that various changes and modifications may be madewithout departing from the scope and spirit of the invention.

It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

The foregoing description of various embodiments of the invention hasbeen presented for purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the preciseembodiments disclosed. Numerous modifications or variations are possiblein light of the above teachings. The embodiments discussed were chosenand described to provide the best illustration of the principles of theinvention and its practical application to thereby enable one ofordinary skill in the art to utilize the invention in variousembodiments and with various modifications as are suited to theparticular use contemplated. All such modifications and variations arewithin the scope of the invention as determined by the appended claimswhen interpreted in accordance with the breadth to which they arefairly, legally, and equitably entitled.

1. A hardware-in-the-loop simulation system to verify computer vision,the system comprising: a virtual reality imaging unit that generates avirtual scene and an observed object using a virtual reality software,wherein virtual scene images of the scene at different viewpoints areobtained; a projector that projects the virtual scene images generatedby the virtual reality software onto a screen; and a camera that shootsthe virtual scene images projected by said projector on the screen,wherein the direction of the camera is controlled by a pan-tilt, whereinthe camera is fixed on the pan-tilt, and wherein a position andorientation of the projector and the pan-tilt are fixed to the screen;wherein the virtual reality imaging unit comprises a first computer andwherein said projector connects to the virtual reality imaging unit; andwherein the camera sends image data gathered while shooting theprojected scene images to a second computer by a frame grabber board andthe second computer is configured to control the pan-tilt to select aproper direction for the camera to shoot the virtual scene imagesprojected onto the screen.
 2. A hardware-in-the-loop simulation methodfor computer vision, the method comprising: generating virtual objectsor scenes with a first computer using virtual reality software installedon the first computer; setting parameters for the virtual realitysoftware to acquire an image of the virtual objects or scenes;projecting the image onto a screen; taking the image of the virtualobjects or scenes on the screen using an actual camera to acquire aresult of a simulation; calculating parameters of a virtual camera witha second computer, and calibrating models for the actual camera and aprojector for a high-accuracy parameter estimation with the secondcomputer; and achieving computer vision experiments based on the resultof the simulation with the second computer.
 3. The hardware-in-the-loopsimulation method according to claim 2, wherein the step of setting theparameters for the virtual reality software further comprise: setting avertical field of view angle in degrees, an aspect ratio of a width toheight of a viewport, a distance from the viewpoint to near and farclipping planes, coordinates in a world coordinate system of theviewpoint, and yaw, pitch and roll angles.
 4. The hardware-in-the-loopsimulation method according to claim 2, wherein the step of calculatingthe parameters of the virtual camera is relative to the virtual realitysoftware parameters; the intrinsic parameters of the virtual camera arecalculated according to parameters of a projection transformation of thevirtual reality software, and extrinsic parameters of the virtual cameraare calculated according to parameters of a model view transformation ofthe virtual reality software.
 5. The hardware-in-the-loop simulationmethod according to claim 2, wherein in the step of calibrating theactual camera and projector models, the projector model and the actualcamera model are regarded as a whole module for calibration.
 6. Thehardware-in-the-loop simulation method according to claim 5, wherein anonlinear camera calibration technique is used for high-accuracyparameter estimation of said whole module for calibration, comprising:determining the center of a camera image using a method of varying focallength; achieving lens distortion calibration using a calibrationapproach based on cross ratio invariability; calculating a coordinate ofa corner of a calibration pattern in a camera coordinate system based onthe center of the camera image and the lens distortion, according to animage coordinate in pixel; setting up linear equation groups accordingto a linear transformation model from the camera coordinate system to aprojector computer image coordinate system; wherein a lineartransformation matrix is estimated using a least-squares method toachieve the parameter estimation.